Concept Drift Adaptation Methods under the Deep Learning Framework: A Literature Review

With the advent of the fourth industrial revolution, data-driven decision making has also become an integral part of decision making. At the same time, deep learning is one of the core technologies of the fourth industrial revolution that have become vital in decision making. However, in the era of epidemics and big data, the volume of data has increased dramatically while the sources have become progressively more complex, making data distribution highly susceptible to change. These situations can easily lead to concept drift, which directly affects the effectiveness of prediction models. How to cope with such complex situations and make timely and accurate decisions from multiple perspectives is a challenging research issue. To address this challenge, we summarize concept drift adaptation methods under the deep learning framework, which is beneficial to help decision makers make better decisions and analyze the causes of concept drift. First, we provide an overall introduction to concept drift, including the definition, causes, types, and process of concept drift adaptation methods under the deep learning framework. Second, we summarize concept drift adaptation methods in terms of discriminative learning, generative learning, hybrid learning, and others. For each aspect, we elaborate on the update modes, detection modes, and adaptation drift types of concept drift adaptation methods. In addition, we briefly describe the characteristics and application fields of deep learning algorithms using concept drift adaptation methods. Finally, we summarize common datasets and evaluation metrics and present future directions.

STAD: State-Transition-Aware Anomaly Detection Under Concept Drifts

The detection of temporal abnormal patterns over streaming data is challenging due to volatile data properties and lacking real-time labels. The abnormal patterns are usually hidden in the temporal context, which can not be detected by evaluating single points. Furthermore, the normal state evolves over time due to concept drift. A single model does not fit all data over time. Autoencoders are recently applied for unsupervised anomaly detection. However, they usually get expired and invalid after distributional drifts in the data stream. In this paper, we propose an autoencoder-based approach (STAD) for anomaly detection under concept drift. In particular, we use a state-transition-based model to map different data distributions in each period of the data stream into states, thereby addressing the model adaptation problem in an interpretable way. We empirically demonstrate the state transition process and evaluate the anomaly detection performance on the Covid-19 dataset of Germany. © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org)

Is My Model Up-to-date? Detecting CoViD-19 Variants by Machine Learning

Machine learning extracts models from huge quantities of data. Models trained and validated over past data can be deployed in making forecasts as well as in classifying new incoming data. The real world which generates data may change over time, making the deployed model an obsolete one. To preserve the quality of the currently deployed model, continuous machine learning is required. Our approach retrospectively evaluates in an online fashion the behaviour of the currently deployed model. A drift detector detects any performance slump, and, in case, can replace the previous model with an up-to-date one. The approach experiments on a dataset of 8642 hematochemical examinations from hospitalized patients gathered over 6 months: the outcome of the model predicts the RT-PCR test result about CoViD-19. The method reached an area under the curve (AUC) of 0.794, 6% better than offline and 5% better than standard online-binary classification techniques. © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org)

Is My Model Up-to-date? Detecting CoViD-19 Variants by Machine Learning

Machine learning extracts models from huge quantities of data. Models trained and validated over past data can be deployed in making forecasts as well as in classifying new incoming data. The real world which generates data may change over time, making the deployed model an obsolete one. To preserve the quality of the currently deployed model, continuous machine learning is required. Our approach retrospectively evaluates in an online fashion the behaviour of the currently deployed model. A drift detector detects any performance slump, and, in case, can replace the previous model with an up-to-date one. The approach experiments on a dataset of 8642 hematochemical examinations from hospitalized patients gathered over 6 months: the outcome of the model predicts the RT-PCR test result about CoViD-19. The method reached an area under the curve (AUC) of 0.794, 6% better than offline and 5% better than standard online-binary classification techniques. © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org)

Ensemble Learning for Interpretable Concept Drift and Its Application to Drug Recommendation

During the COVID-19 epidemic, the online prescription pattern of Internet healthcare pro-vides guarantee for the patients with chronic diseases and reduces the risk of cross-infection, but it also raises the burden of decision-making for doctors. Online drug recommendation system can effectively assist doctors by analysing the electronic medical records (EMR) of patients. Unlike commercial recommendations, the accuracy of drug recommendations should be very high due to their relevance to patient health. Besides, concept drift may occur in the drug treatment data streams, handling drift and location drift causes is critical to the accuracy and reliability of the rec-ommended results. This paper proposes a multi-model fusion online drug recommendation system based on the association of drug and pathological features with online-nearline-offline architecture.The system transforms drug recommendation into pattern classification and adopts interpretable concept drift detection and adaptive ensemble classification algorithms. We apply the system to the Percutaneous Coronary Intervention (PCI) treatment process. The experiment results show our system performs nearly as good as doctors, the accuracy is close to 100%.

Forecasting patient flows with pandemic induced concept drift using explainable machine learning.

Accurately forecasting patient arrivals at Urgent Care Clinics (UCCs) and Emergency Departments (EDs) is important for effective resourcing and patient care. However, correctly estimating patient flows is not straightforward since it depends on many drivers. The predictability of patient arrivals has recently been further complicated by the COVID-19 pandemic conditions and the resulting lockdowns. This study investigates how a suite of novel quasi-real-time variables like Google search terms, pedestrian traffic, the prevailing incidence levels of influenza, as well as the COVID-19 Alert Level indicators can both generally improve the forecasting models of patient flows and effectively adapt the models to the unfolding disruptions of pandemic conditions. This research also uniquely contributes to the body of work in this domain by employing tools from the eXplainable AI field to investigate more deeply the internal mechanics of the models than has previously been done. The Voting ensemble-based method combining machine learning and statistical techniques was the most reliable in our experiments. Our study showed that the prevailing COVID-19 Alert Level feature together with Google search terms and pedestrian traffic were effective at producing generalisable forecasts. The implications of this study are that proxy variables can effectively augment standard autoregressive features to ensure accurate forecasting of patient flows. The experiments showed that the proposed features are potentially effective model inputs for preserving forecast accuracies in the event of future pandemic outbreaks.

Immunosuppression as a Hub for SARS-CoV-2 Mutational Drift.

The clinical course of coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2), is largely determined by host factors, with a wide range of outcomes. Despite an extensive vaccination campaign and high rates of infection worldwide, the pandemic persists, adapting to overcome antiviral immunity acquired through prior exposure. The source of many such major adaptations is variants of concern (VOCs), novel SARS-CoV-2 variants produced by extraordinary evolutionary leaps whose origins remain mostly unknown. In this study, we tested the influence of factors on the evolutionary course of SARS-CoV-2. Electronic health records of individuals infected with SARS-CoV-2 were paired to viral whole-genome sequences to assess the effects of host clinical parameters and immunity on the intra-host evolution of SARS-CoV-2. We found slight, albeit significant, differences in SARS-CoV-2 intra-host diversity, which depended on host parameters such as vaccination status and smoking. Only one viral genome had significant alterations as a result of host parameters; it was found in an immunocompromised, chronically infected woman in her 70s. We highlight the unusual viral genome obtained from this woman, which had an accelerated mutational rate and an excess of rare mutations, including near-complete truncating of the accessory protein ORF3a. Our findings suggest that the evolutionary capacity of SARS-CoV-2 during acute infection is limited and mostly unaffected by host characteristics. Significant viral evolution is seemingly exclusive to a small subset of COVID-19 cases, which typically prolong infections in immunocompromised patients. In these rare cases, SARS-CoV-2 genomes accumulate many impactful and potentially adaptive mutations; however, the transmissibility of such viruses remains unclear.

Has the pandemic resulted in a renewed and improved focus on heath inequalities in England? A discourse analysis of the framing of health inequalities in national policy.

Objectives: The COVID-19 pandemic rapidly exacerbated health inequalities in England. Policy makers sought to ameliorate its impact. This paper aims to identify how health inequalities were framed in national policy documents published in England during the pandemic and how this impacts the framing of policy solutions. Study design: Discourse analysis of selected national policy documents. Methods: First, we identified relevant national policy documents through a broad search and eligibility criteria to identify illustrative policy documents. Second, we undertook a discourse analysis to understand the framing and constitution of health inequalities and consequent solutions within them. Third, we used existing health inequalities literature to critique the findings. Results: Based on analysis of six documents, we found evidence of the idea of lifestyle drift with a marked disjunction between the acknowledgement of the wider determinants of heath and the policy solutions advocated. The target population for interventions is predominantly the worst off, rather than the whole social gradient. Repeated appeals to behaviour change indicate an inherent individualist epistemology. Responsibility and accountability for health inequalities appears delegated locally without the power and resource required to deliver. Conclusion: Policy solutions are unlikely to address health inequalities. This could be done though through (i) shifting interventions towards structural factors and wider determinants of health, (ii) a positive vision of a health equitable society, (iii) a proportional universalism in approach and (iv) a delegation of power and resource alongside responsibility for delivering on health inequalities. These possibilities currently remain outside of the policy language of health inequalities.

Concept Drift Aware Analysis of Learning Engagement During COVID-19 Pandemic Using Adaptive Windowing

Change is the only thing in the real world which has been known to last forever. It takes various forms and progressions, ranging from gradual in some cases and abrupt in the others to even constantly incremental in yet other cases like ageing. Machine learning (ML) algorithms, in its simplest definitions, use the statistical analysis of static past data records to make predictions about the future and have reached a fair amount of accuracy on diverse data sets across different application domains. There exists an inherent contradictory friction between real life analysis and machine learning models based on above definitions, and it gets compounded while capturing the ever-changing data from streaming sources. Concept drift is a principle used for description of unpredictable variations in streaming data sourced from the real world through a given time period. The drift phenomenon occurring even in a single feature, if left unaddressed leads to silent decay and can play havoc with the accuracy of a previously accurate ML model. With increasing prevalence and scale of real-world deployments of ML analytics, models cannot remain invariant to instability of data distributions and must adapt to concept drift. We analyse the occurrence and effect of concept drift in the COVID-19 online education data sourced from LearnPlatform edtech Company in this paper. The data set has almost 20 million entries related to engagement index and can be fairly assumed to be big data for processing purposes. A comparative case analysis for the accuracy of concept drift aware modelling using adaptive windowing (ADWIN) vis-a-vis the basic ML counterpart to predict the student engagement based on digital connectivity and education technology has been carried out for the study. © 2023, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

Fund style drift and stock price crash risk – analysis of the mediating effect based on corporate financial risk

PurposeThis paper aims to identify the direct impact of fund style drift on the risk of stock price collapse and the intermediary mechanism of financial risk, so as to better protect the interests of minority investors.Design/methodology/approachThis paper takes all the non-financial companies on the Chinese Growth Enterprise Market from 2011 to 2020 as study object and selects securities investment funds of their top ten circulation stocks to study the relationship between fund style drift and stock price crash risk.FindingsFund style drift is likely to add stock price crash risk. Financial risk is positively correlated with stock price crash risk. Fund style drift affects stock price crash risk via the mediating effect of financial risk, and fund style drift and financial risk have a marked impact on the stock price crash risk of non-state enterprises, yet a non-significant impact on that of state-owned enterprises.Originality/valueThis paper links fund style drift with stock price crash risk in an exploratory manner and enriches the study perspectives of relationship between institutional investors' behaviors and stock price crash risk, thus enjoying certain academic value. On the one hand, it furnishes a new approach to the academic frontier issue concerning financial risk and stock price crash risk, and proves that financial risk is positively correlated with stock price crash risk. On the other hand, it regards financial risk as a mediating variable of fund style drift for stock price crash risk and further explores different influencing mechanism of institutional investors' behaviors.

Polymorphism and Selection Pressure of SARS-CoV-2 Vaccine and Diagnostic Antigens: Implications for Immune Evasion and Serologic Diagnostic Performance.

The ongoing SARS-CoV-2 pandemic has triggered multiple efforts for serological tests and vaccine development. Most of these tests and vaccines are based on the Spike glycoprotein (S) or the Nucleocapsid (N) viral protein. Conservation of these antigens among viral strains is critical to ensure optimum diagnostic test performance and broad protective efficacy, respectively. We assessed N and S antigen diversity from 17,853 SARS-CoV-2 genome sequences and evaluated selection pressure. Up to 6-7 incipient phylogenetic clades were identified for both antigens, confirming early variants of the S antigen and identifying new ones. Significant diversifying selection was detected at multiple sites for both antigens. Some sequence variants have already spread in multiple regions, in spite of their low frequency. In conclusion, the N and S antigens of SARS-CoV-2 are well-conserved antigens, but new clades are emerging and may need to be included in future diagnostic and vaccine formulations.

Detecting Drift in Healthcare AI Models Based on Data Availability

There is an increasing interest in the use of AI in healthcare due to its potential for diagnosis or disease prediction. However, healthcare data is not static and is likely to change over time leading a non-adaptive model to poor decision-making. The need of a drift detector in the overall learning framework is therefore essential to guarantee reliable products on the market. Most drift detection algorithms consider that ground truth labels are available immediately after prediction since these methods often work by monitoring the model performance. However, especially in real-world clinical contexts, this is not always the case as collecting labels is often more time consuming as requiring experts' input. This paper investigates methodologies to address drift detection depending on which information is available during the monitoring process. We explore the topic within a regulatory standpoint, showing challenges and approaches to monitoring algorithms in healthcare with subsequent batch updates of data. This paper explores three different aspects of drift detection: drift based on performance (when labels are available), drift based on model structure (indicating causes of drift) and drift based on change in underlying data characteristics (distribution and correlation) when labels are not available. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Assessing the effects of data drift on the performance of machine learning models used in clinical sepsis prediction.

BACKGROUND: Data drift can negatively impact the performance of machine learning algorithms (MLAs) that were trained on historical data. As such, MLAs should be continuously monitored and tuned to overcome the systematic changes that occur in the distribution of data. In this paper, we study the extent of data drift and provide insights about its characteristics for sepsis onset prediction. This study will help elucidate the nature of data drift for prediction of sepsis and similar diseases. This may aid with the development of more effective patient monitoring systems that can stratify risk for dynamic disease states in hospitals. METHODS: We devise a series of simulations that measure the effects of data drift in patients with sepsis, using electronic health records (EHR). We simulate multiple scenarios in which data drift may occur, namely the change in the distribution of the predictor variables (covariate shift), the change in the statistical relationship between the predictors and the target (concept shift), and the occurrence of a major healthcare event (major event) such as the COVID-19 pandemic. We measure the impact of data drift on model performances, identify the circumstances that necessitate model retraining, and compare the effects of different retraining methodologies and model architecture on the outcomes. We present the results for two different MLAs, eXtreme Gradient Boosting (XGB) and Recurrent Neural Network (RNN). RESULTS: Our results show that the properly retrained XGB models outperform the baseline models in all simulation scenarios, hence signifying the existence of data drift. In the major event scenario, the area under the receiver operating characteristic curve (AUROC) at the end of the simulation period is 0.811 for the baseline XGB model and 0.868 for the retrained XGB model. In the covariate shift scenario, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.853 and 0.874 respectively. In the concept shift scenario and under the mixed labeling method, the retrained XGB models perform worse than the baseline model for most simulation steps. However, under the full relabeling method, the AUROC at the end of the simulation period for the baseline and retrained XGB models is 0.852 and 0.877 respectively. The results for the RNN models were mixed, suggesting that retraining based on a fixed network architecture may be inadequate for an RNN. We also present the results in the form of other performance metrics such as the ratio of observed to expected probabilities (calibration) and the normalized rate of positive predictive values (PPV) by prevalence, referred to as lift, at a sensitivity of 0.8. CONCLUSION: Our simulations reveal that retraining periods of a couple of months or using several thousand patients are likely to be adequate to monitor machine learning models that predict sepsis. This indicates that a machine learning system for sepsis prediction will probably need less infrastructure for performance monitoring and retraining compared to other applications in which data drift is more frequent and continuous. Our results also show that in the event of a concept shift, a full overhaul of the sepsis prediction model may be necessary because it indicates a discrete change in the definition of sepsis labels, and mixing the labels for the sake of incremental training may not produce the desired results.

Role of epigenetics in the clinical evolution of COVID-19 disease. Epigenome-wide association study identifies markers of severe outcome.

BACKGROUND: COVID-19 has a wide spectrum of clinical manifestations and given its impact on morbidity and mortality, there is an unmet medical need to discover endogenous cellular and molecular biomarkers that predict the expected clinical course of the disease. Recently, epigenetics and especially DNA methylation have been pointed out as a promising tool for outcome prediction in several diseases. METHODS AND RESULTS: Using the Illumina Infinium Methylation EPIC BeadChip850K, we investigated genome-wide differences in DNA methylation in an Italian Cohort of patients with comorbidities and compared severe (n = 64) and mild (123) prognosis. Results showed that the epigenetic signature, already present at the time of Hospital admission, can significantly predict risk of severe outcomes. Further analyses provided evidence of an association between age acceleration and a severe prognosis after COVID-19 infection. The burden of Stochastic Epigenetic Mutation (SEMs) has been significantly increased in patients with poor prognosis. Results have been replicated in silico considering COVID-19 negative subjects and available previously published datasets. CONCLUSIONS: Using original methylation data and taking advantage of already published datasets, we confirmed in the blood that epigenetics is actively involved in immune response after COVID-19 infection, allowing the identification of a specific signature able to discriminate the disease evolution. Furthermore, the study showed that epigenetic drift and age acceleration are associated with severe prognosis. All these findings prove that host epigenetics undergoes notable and specific rearrangements to respond to COVID-19 infection which can be used for a personalized, timely, and targeted management of COVID-19 patients during the first stages of hospitalization.

Antigenic characterization of influenza and SARS-CoV-2 viruses.

Antigenic characterization of emerging and re-emerging viruses is necessary for the prevention of and response to outbreaks, evaluation of infection mechanisms, understanding of virus evolution, and selection of strains for vaccine development. Primary analytic methods, including enzyme-linked immunosorbent/lectin assays, hemagglutination inhibition, neuraminidase inhibition, micro-neutralization assays, and antigenic cartography, have been widely used in the field of influenza research. These techniques have been improved upon over time for increased analytical capacity, and some have been mobilized for the rapid characterization of the SARS-CoV-2 virus as well as its variants, facilitating the development of highly effective vaccines within 1 year of the initially reported outbreak. While great strides have been made for evaluating the antigenic properties of these viruses, multiple challenges prevent efficient vaccine strain selection and accurate assessment. For influenza, these barriers include the requirement for a large virus quantity to perform the assays, more than what can typically be provided by the clinical samples alone, cell- or egg-adapted mutations that can cause antigenic mismatch between the vaccine strain and circulating viruses, and up to a 6-month duration of vaccine development after vaccine strain selection, which allows viruses to continue evolving with potential for antigenic drift and, thus, antigenic mismatch between the vaccine strain and the emerging epidemic strain. SARS-CoV-2 characterization has faced similar challenges with the additional barrier of the need for facilities with high biosafety levels due to its infectious nature. In this study, we review the primary analytic methods used for antigenic characterization of influenza and SARS-CoV-2 and discuss the barriers of these methods and current developments for addressing these challenges.

Vaccine effectiveness against influenza A(H3N2)-associated hospitalized illness, United States, 2022.

BACKGROUND: The COVID-19 pandemic was associated with historically low influenza circulation during the 2020-2021 season, followed by increase in influenza circulation during the 2021-2022 US season. The 2a.2 subgroup of the influenza A(H3N2) 3C.2a1b subclade that predominated was antigenically different from the vaccine strain. METHODS: To understand the effectiveness of the 2021-2022 vaccine against hospitalized influenza illness, a multi-state sentinel surveillance network enrolled adults aged ≥18 years hospitalized with acute respiratory illness (ARI) and tested for influenza by a molecular assay. Using the test-negative design, vaccine effectiveness (VE) was measured by comparing the odds of current season influenza vaccination in influenza-positive case-patients and influenza-negative, SARS-CoV-2-negative controls, adjusting for confounders. A separate analysis was performed to illustrate bias introduced by including SARS-CoV-2 positive controls. RESULTS: A total of 2334 patients, including 295 influenza cases (47% vaccinated), 1175 influenza- and SARS-CoV-2 negative controls (53% vaccinated), and 864 influenza-negative and SARS-CoV-2 positive controls (49% vaccinated), were analyzed. Influenza VE was 26% (95%CI: -14 to 52%) among adults aged 18-64 years, -3% (95%CI: -54 to 31%) among adults aged ≥65 years, and 50% (95%CI: 15 to 71%) among adults 18-64 years without immunocompromising conditions. Estimated VE decreased with inclusion of SARS-CoV-2-positive controls. CONCLUSIONS: During a season where influenza A(H3N2) was antigenically different from the vaccine virus, vaccination was associated with a reduced risk of influenza hospitalization in younger immunocompetent adults. However, vaccination did not provide protection in adults ≥65 years of age. Improvements in vaccines, antivirals, and prevention strategies are warranted.

Comparing airborne infectious aerosol exposures in sparsely occupied large spaces utilizing large-diameter ceiling fans

In sparsely occupied large industrial and commercial buildings, large-diameter ceiling fans1 (LDCFs) are commonly utilized for comfort cooling and destratification;however, a limited number of studies were conducted to guide the operation of these devices during the COVID-19 pandemic. This study conducted 223 parametrical computational-fluid-dynamics (CFD) simulations of LDCFs in the U.S. Department of Energy warehouse reference building to compare the impacts of fan operations, index-person, and worker-packing-line locations on airborne exposures to infectious aerosols under both summer and winter conditions. The steady-state airflow fields were modeled while transient exposures to particles of varying sizes (0.5–10 μm) were evaluated over an 8-h period. Both the airflow and aerosol models were validated by measurement data from the literature. It was found that it is preferable to create a breeze from LDCFs for increased airborne dilution into a sparsely occupied large warehouse, which is more similar to an outdoor scenario than a typical indoor scenario. Operation of fans at the highest feasible speed while maintaining thermal-comfort requirements consistently outperformed the other options in terms of airborne exposures. There is no substantial evidence that fan reversal is beneficial in the current large space of interest. Reversal flow direction to create upward flows at higher fan speeds generally reduced performance compared with downward flows, as there was less airflow through the fan blades at the same rotational speed. Reversing flow at lower fan speeds decreased airflow speeds and dilution in the space and, thus, increased whole-warehouse concentrations. © 2023 Elsevier Ltd

Bayesian nonlinear expectation for time series modelling and its application to Bitcoin.

This paper proposes a two-stage approach to parametric nonlinear time series modelling in discrete time with the objective of incorporating uncertainty or misspecification in the conditional mean and volatility. At the first stage, a reference or approximating time series model is specified and estimated. At the second stage, Bayesian nonlinear expectations are introduced to incorporate model uncertainty or misspecification in prediction via specifying a family of alternative models. The Bayesian nonlinear expectations for prediction are constructed from closed-form Bayesian credible intervals evaluated using conjugate priors and residuals of the estimated approximating model. Using real Bitcoin data including some periods of Covid 19, applications of the proposed method to forecasting and risk evaluation of Bitcoin are discussed via three major parametric nonlinear time series models, namely the self-exciting threshold autoregressive model, the generalized autoregressive conditional heteroscedasticity model and the stochastic volatility model. Supplementary Information: The online version contains supplementary material available at 10.1007/s00181-022-02255-z.

A brute force tuning of training length for concept drift

We present a brute-force approach to analyze the concept drift behind time sequence data. This approach, named SELECT, searches for the optimal length of training data to minimize error metrics. In other words, SELECT searches for the start point of a new concept from the input sequence. Unlike many related methods, SELECT does not require a pre-specified error threshold to detect drift. In addition, the visual analysis obtained from SELECT enables us to understand how significant a drift has occurred. We test SELECT on two real-world datasets, stock price and COVID-19 infection data. The experimental results show that SELECT can improve the model performance of both datasets. In addition, the visual analysis shows the points of significant drifts, e.g., Lehman's collapse in stock price data and the spread of variants in COVID-19 data. These results show the effectiveness of the brute-force approach in analyzing concept drift. Copyright © 2023 Uchida and Yoshida.

Multisite Evaluation of Prediction Models for Emergency Department Crowding Before and During the COVID-19 Pandemic.

OBJECTIVE: To develop a machine learning framework to forecast emergency department (ED) crowding and to evaluate model performance under spatial and temporal data drift. MATERIALS AND METHODS: We obtained four datasets, identified by the location: 1-large academic hospital and 2-rural hospital, and time period: pre-COVID (Jan 1, 2019-Feb 1, 2020) and COVID-era (May 15, 2020-Feb 1, 2021). Our primary target was a binary outcome that is equal to 1 if the number of patients with acute respiratory illness that were ED boarding for more than four hours was above a prescribed historical percentile. We trained a random forest and used the area under the curve (AUC) to evaluate out-of-sample performance for two experiments: 1) we evaluated the impact of sudden temporal drift by training models using pre-COVID data and testing them during the COVID-era, 2) we evaluated the impact of spatial drift by testing models trained at Location 1 on data from Location 2, and vice versa. RESULTS: The baseline AUC values for ED boarding ranged from 0.54 (pre-COVID at Location 2) to 0.81 (COVID-era at Location 1). Models trained with pre-COVID data performed similarly to COVID-era models (0.82 vs. 0.78 at Location 1). Models that were transferred from Location 2 to Location 1 performed worse than models trained at Location 1 (0.51 vs. 0.78). DISCUSSION AND CONCLUSION: Our results demonstrate that ED boarding is a predictable metric for ED crowding, models were not significantly impacted by temporal data drift, and any attempts at implementation must consider spatial data drift.